Scaling of scoring rules
A talk about scaling of scoring rules (model evaluation)
Abstract
Averages of proper scoring rules are often used to rank probabilistic forecasts. In many cases, the individual observations and their predictive distributions in these averages have variable scale (variance). I will show that some of the most popular proper scoring rules, such as the continuous ranked probability score (CRPS), up-weight observations with large uncertainty which can lead to unintuitive rankings. We have developed a new scoring rule, scaled CRPS (SCRPS), this new proper scoring rule is locally scale invariant and therefore works in the case of varying uncertainty. I will demostrate this how this affects model selection through parameter estimation in spatial statitics.
Citation
BibTeX citation:
@unpublished{wallin2020,
author = {Wallin, Jonas and Bolin, David},
title = {Scaling of Scoring Rules},
date = {2020-06-08},
url = {https://www.cirm-math.com/cirm-virtual-event-2146.html},
langid = {en},
abstract = {Averages of proper scoring rules are often used to rank
probabilistic forecasts. In many cases, the individual observations
and their predictive distributions in these averages have variable
scale (variance). I will show that some of the most popular proper
scoring rules, such as the continuous ranked probability score
(CRPS), up-weight observations with large uncertainty which can lead
to unintuitive rankings. We have developed a new scoring rule,
scaled CRPS (SCRPS), this new proper scoring rule is locally scale
invariant and therefore works in the case of varying uncertainty. I
will demostrate this how this affects model selection through
parameter estimation in spatial statitics.}
}
For attribution, please cite this work as:
Wallin, Jonas, and David Bolin. 2020. “Scaling of Scoring
Rules.” Online, Online, June 8. https://www.cirm-math.com/cirm-virtual-event-2146.html.